Liquid droplet path prediction

ABSTRACT

A system includes a processor and a memory storing instructions executable by the processor to predict a path of a liquid droplet on a surface, and then, actuate one or more vehicle components based on the path.

BACKGROUND

A vehicle may use image data from an optical sensor for operation.Liquid droplets on a surface in a field of view of the optical sensormay reduce a usefulness of the image data for operating the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a vehicle having a system for predictinga path of a liquid droplet.

FIG. 2 is a section view of a portion of the vehicle including awindshield and an optical sensor.

FIG. 3 is a block diagram of components of the system and the vehicle.

FIG. 4 illustrates an example Deep Neural Network (DNN).

FIG. 5 is an illustration of an example image captured by the opticalsensor.

FIG. 6 is a flow chart illustrating a process for controlling thevehicle having the system for prediction a path of a liquid droplet.

DETAILED DESCRIPTION Introduction

A system includes a processor and a memory storing instructionsexecutable by the processor to predict a path of a liquid droplet on asurface, and then, actuate one or more vehicle components based on thepath.

The instructions may further include instructions to determine that thepath is from a low priority area on the surface to a high priority areaon the surface, where the high priority area has a higher probability ofcontaining detectable objects that may interfere with operation of avehicle as compared to the low priority area.

The instructions may further include instructions to predict the pathbased on at least one of a vehicle velocity, a wind velocity, an inclineangle of the surface, a vehicle acceleration, a size of the liquiddroplet, and a hydrophobicity of the surface.

The instructions may further include instructions to predict the pathbased on stored data indicating one or more previous positions of theliquid droplet.

The instructions may further include instructions to actuate the one ormore vehicle components by actuating a cleaning system.

The instructions may further include instructions to predict a secondpath of the liquid droplet after actuating the cleaning system.

The instructions may further include instructions to actuate the one ormore vehicle components by actuating at least one of a steering system,a braking system, and a propulsion system.

The instructions may further include instructions to wait an amount oftime after predicting the path, and then update one or more parametersof the prediction based on a comparison of a position of the liquiddroplet with the path.

The system may include an optical sensor defining a field of view and incommunication with the processor, wherein the surface is in the field ofview.

The instructions may further include instructions to identify an amountof time for the liquid droplet to reach a future position, and todetermine the amount of time to reach the future position is greaterthan a threshold amount of time.

The surface may be on one of a lens and a windshield.

A method includes predicting a path of a liquid droplet on a surface,and then, actuating one or more vehicle components based on the path.

The path may be predicted based on at least one of a vehicle velocity, awind velocity, an incline angle of the surface, a vehicle acceleration,a size of the liquid droplet, and a hydrophobicity of the surface.

The method may include waiting an amount of time after predicting thepath, and then updating one or more parameters of the prediction basedon a comparison of a position of the liquid droplet with the path.

The method may include identifying an amount of time for the liquiddroplet to reach a future position, and to determine the amount of timeto reach the future position is greater than a threshold amount of time.

Actuating one or more vehicle components may include actuating acleaning system, and the method may further include predicting a secondpath of the liquid droplet after actuating the cleaning system.

The path may be predicted based on stored data indicating one or moreprevious positions of the liquid droplet.

A system includes means for detecting a liquid droplet on a surface. Thesystem includes means for predicting a path of the liquid droplet on thesurface. The system includes means for moving the liquid dropletrelative to the surface based on the path.

The system may include means for navigating a vehicle based on the path.

The system may include means for determining that the path is from a lowpriority area on the surface to a high priority area on the surface,wherein the high priority area has a higher probability of containingdetectable objects that may interfere with operation of a vehicle ascompared to the low priority area.

With reference to the Figures, wherein like numerals indicate like partsthroughout the several views, a system 10 for a vehicle 12 includesmeans for detecting a liquid droplet 14 on a surface 16 a, 16 b. Anexample means for detecting the liquid droplet 14 includes an opticalsensor 18 directed at a surface 16 a on a windshield 20 or a surface 16b on a lens 22. The system 10 includes means for predicting a path 24 ofthe liquid droplet 14 on the surface 16 a, 16 b. An example means forpredicting the path includes a computer 26 having a processor and amemory storing instructions executable by the processor to predict thepath 24 of the liquid droplet 14 on the surface 16 a, 16 b, and then,actuate one or more vehicle components based on the path 24. The system10 includes means for moving the liquid droplet 14 relative to thesurface 16 a, 16 b based on the path 24. An example means for moving theliquid droplet 14 include a cleaning system 28 in communication with thecomputer 26.

As used herein, vehicle components are systems, assemblies,sub-assemblies, and/or other structures of the vehicle 12 actuatable bythe computer 26 to perform a physical function, e.g., actuate thecleaning system 28, a propulsion system 30, a steering system 32, and/ora braking system 34.

Predicting the path 24 of the liquid droplet 14 aids in autonomous orsemi-autonomous operation of the vehicle 12 by predicting when adetected object 36 may be obscured by the liquid droplet 14, and byreducing interference of the liquid droplet 14 with a view of an areathat has a relatively higher probability of including detectable objects36. For example, the computer 26 may operate the vehicle 12 such thatthe path 24 is repositioned relative to the higher probability area.Predicting the path 24 of the liquid droplet 14 permits efficient use ofvehicle resources when clearing the surface 16 a, 16 b. For example, thecomputer 26 may refraining from actuating the cleaning system 28 whenthe path 24 indicates that liquid droplet 14 will not interfere withdata collected by the optical sensor 18 in the higher probability areas.

Apparatus

The vehicle 12 may be any type of passenger or commercial automobilesuch as a car, a truck, a sport utility vehicle, a crossover vehicle, avan, a minivan, a taxi, a bus, etc.

The windshield 20 protects an interior of the vehicle 12, e.g., fromwind, precipitation, debris, etc. The windshield 20 is transparent,e.g., such that occupants of the vehicle 12 may see therethrough. Thewindshield 20 may be supported by the vehicle 12 at a forward end of apassenger cabin, a rearward end of the passenger cabin, etc. The surface16 a may be away from the passenger cabin, e.g., relative to thewindshield 20. In other words, the surface 16 a is typically outside thepassenger cabin.

The optical sensor 18 detects light. The optical sensor 18 may be ascanning laser range finder, a light detection and ranging (LIDAR)device, an image processing sensor such as a camera, or any other sensorthat detects light. One more or more optical sensors 18 may be supportedby the vehicle 12, e.g. stereo camera pair. For example, one of theoptical sensors 18 may detect light through the windshield 20 andanother of the optical sensors 18 may detect light through the lens 22.In another example, the optical sensor 18may be capable of measuringboth the reflectivity of objects and their distances, e.g. a time offlight camera. The optical sensor 18may be capable of detecting on ormore wavelengths, such as red, blue, green, visible light, nearinfrared, etc. The optical sensors 18 may be placed in proximity andshare the same exterior facing optical surface, e.g. windshield,allowing one camera image to predict a future occlusion in anothercamera of the same water droplet passing along the respective field ofview of each optical sensor 18.

The optical sensor 18 has a field of view 38. The field of view 38 is avolume relative to the optical sensor 18 from which light is detectableby the optical sensor 18. In other words, light generated by, and/orreflected off, an object within the field of view 38, and towards theoptical sensor 18, is detectable by the optical sensor 18, provided suchlight is not blocked before reaching the optical sensor 18. The surface16 a, 16 b is in the field of view 38, e.g., depending on a location ofthe optical sensor 18.

The optical sensor 18 generates image data based on light detected fromwithin the field of view 38. The image data indicates a detected imagewith a two-dimensional array of pixels, e.g., a grid having rows andcolumns of pixels. Each pixel may indicate a color, a brightness, a hue,etc., of light detected from a specific portion of the field of view 38.An illustration of an example image that may be indicated by image datafrom the optical sensor 18 is shown in FIG. 5.

The vehicle 12 may include other sensors 40. The sensors 40 may detectinternal states of the vehicle 12, for example, wheel speed, wheelorientation, and engine and transmission variables. The sensors 40 maydetect the position or orientation of the vehicle, for example, globalpositioning system (GPS) sensors; accelerometers such as piezo-electricor microelectromechanical systems (MEMS) sensors; gyroscopes such asrate, ring laser, or fiber-optic gyroscopes; inertial measurements units(IMU); and magnetometers. The sensors 40 may detect the external world,for example, radar sensors, scanning laser range finders, lightdetection and ranging (LIDAR) devices, and image processing sensors suchas cameras. The vehicle 12 may further include communications devices,for example, vehicle-to-infrastructure (V2I) or vehicle-to-vehicle (V2V)devices.

The lens 22 protects the optical sensor 18. The lens 22 may focus lighton the optical sensor 18. The lens 22 may be partially or fullytransparent. The lens 22 may be plastic, glass, etc. The surface 16 bmay be away from the sensor, e.g., relative to a remainder of the lens22. The lens 22 may be rotationally symmetric or non-rotationallysymmetric (a free form lens).

The cleaning system 28 removes objects, such as liquid droplets 14, fromthe surface 16 a of the windshield 20, the surface 16 b of the lens 22,etc., e.g., in response to an instruction from the computer 26. Thecleaning system 28 may include a nozzle 42 directed at the surface 16 a,16 b. The nozzle 42 may be provided with a flow of fluid, e.g., air froma blower, compressed air from a tank, etc., e.g., when the cleaningsystem 28 is actuated to an on state. The air may urge the liquiddroplet 14, e.g., away from the nozzle 42. An additional nozzle 42 mayspray cleaning fluids to assist in the removal of other obstructions.Other techniques may be utilized such as ultrasonic vibration. Air fromthe nozzle 42 may be provided at a specified pressure, rate, duration,etc. The cleaning system 28 may be in communication with the computer26.

The steering system 32 controls a steering angle of wheels of thevehicle 12, e.g., in response to an instruction from the computer 26, inresponse to an operator input, such as to a steering wheel, orcombination of the two such in the case of driver assistivetechnologies. The steering system 32 may be a rack-and-pinion systemwith electric power-assisted steering, a steer-by-wire system, or anyother suitable system for controlling the steering angle of the wheels.The steering system 32 may be in communication with the computer 26.

The braking system 34 resists motion of the vehicle 12 to thereby slowand/or stop the vehicle 12, e.g., in response to an instruction from thecomputer 26 and/or in response to an operator input, such as to a brakepedal. The braking system 34 may include friction brakes such as discbrakes, drum brakes, band brakes, and so on; regenerative brakes; anyother suitable type of brakes; or a combination. The braking system 34may be in communication with the computer 26.

The propulsion system 30 translates energy into motion of the vehicle12, e.g., in response to an instruction from the computer 26 and/or inresponse to an operator input, such as to an accelerator pedal. Forexample, the propulsion system 30 may include a conventional powertrainhaving an internal-combustion engine coupled to a transmission thattransfers rotational motion to wheels; an electric powertrain havingbatteries, an electric motor, and a transmission that transfersrotational motion to the wheels; a hybrid powertrain having elements ofthe conventional powertrain and the electric powertrain; or any othertype of structure for providing motion to the vehicle 12. The propulsionsystem 30 may be in communication with the computer 26.

The vehicle 12 may include a navigation system 44 that can determine alocation of the vehicle 12. The navigation system 44 is implemented viacircuits, chips, or other electronic components. The navigation system44 may be implemented via satellite-based system such as the GlobalPositioning System (GPS). The navigation system 44 may triangulate thelocation of the vehicle 12 based on signals received from varioussatellites in the Earth's orbit. The navigation system 44 is programmedto output signals representing the location of the vehicle 12 to, e.g.,to the computer 26 via a communication network 46. In some instances,the navigation system 44 is programmed to determine a route from thepresent location to a future location. The navigation system 44 mayaccess a virtual map stored in memory of the navigation system 44 and/orcomputer 26, and develop the route according to the virtual map data.The virtual map data may include lane information, including a number oflanes of a road, widths and edges of such lanes, etc.

The communication network 46 includes hardware, such as a communicationbus, for facilitating communication among vehicle components, such asthe computer 26, the propulsion system 30, the steering system 32, thenavigation system 44, the braking system 34, the cleaning system 28, theoptical sensor 18, and other sensors 40. The communication network 46may facilitate wired or wireless communication among the vehiclecomponents in accordance with a number of communication protocols suchas controller area network (CAN), Ethernet, WiFi, Local InterconnectNetwork (LIN), and/or other wired or wireless mechanisms.

The computer 26, implemented via circuits, chips, or other electroniccomponents, is included in the system 10 for carrying out variousoperations, including as described herein. The computer 26 is acomputing device that generally includes a processor and a memory, thememory including one or more forms of computer-readable media, andstoring instructions executable by the processor for performing variousoperations, including as disclosed herein. The memory of the computer 26further generally stores remote data received via various communicationsmechanisms; e.g., the computer 26 is generally configured forcommunications on the communication network 46 or the like, and/or forusing other wired or wireless protocols, e.g., Bluetooth, etc. Thecomputer 26 may also have a connection to an onboard diagnosticsconnector (OBD-II). Via the communication network 46 and/or other wiredor wireless mechanisms, the computer 26 may transmit messages to variousdevices in the vehicle 12 and/or receive messages from the variousdevices, e.g., the steering system 32, the braking system 34, thepropulsion system 30, the optical sensor 18, the cleaning system 28, thenavigation system 44, the sensors 40, etc. Although one computer 26 isshown in FIG. 3 for ease of illustration, it is to be understood thatthe computer 26 could include, and various operations described hereincould be carried out by, one or more computing devices, includingcomputing devices remote from and in communication with the vehicle 12.

The computer 26 may be programmed to, i.e., the memory may storeinstructions executable by the processor to, operate the vehicle 12 inan autonomous mode, a semi-autonomous mode, or a non-autonomous mode.For purposes of this disclosure, the autonomous mode is defined as onein which each of propulsion system 30, the braking system 34, and thesteering system 32 are controlled by the computer 26; in asemi-autonomous mode the computer 26 controls one or two of thepropulsion system 30, the braking system 34, and the steering system 32;in a non-autonomous mode, a human operator controls the propulsionsystem 30, the braking system 34, and the steering system 32.

Operating the vehicle 12, e.g., in the autonomous mode and/orsemi-autonomous mode, may include increasing or decreasing vehiclespeed, changing course heading, etc. The computer 12 may operate thevehicle 12 by transmitting instructions to the steering system 32, thebraking system 34, and the propulsion system 30 based on informationfrom the navigation system 44, the optical sensor 18, and other sensors40. For example, the computer 26 may transmit instructions to thesteering system 32, the braking system 34, and/or the propulsion system30 indicating a change in wheel angle, an increase or decrease inresistance to movement of the vehicle 12, and/or an increase or decreasein power output, respectively. The computer 26 may operate the vehicle12 to navigate one more roads to a destination, to maintain and/orchange lanes of a road, to avoid obstacles, etc.

The computer 26 may be programmed to identify a liquid droplet 14 on thesurface 16 a, 16 b, such as on the surface 16 b on the lens 22, thesurface 16 a on the windshield 20, etc. The computer 26 may identify theliquid droplet 14 based on image data from the optical sensor 18. Forexample, the computer 26 may analyze image data to identify the liquiddroplet 14 according to image recognition techniques, e.g., as areknown.

As one such example, the computer 26 may identify groupings of pixels inimage data having a certain shape, color, brightness, intensitygradient, size, etc. The shape may be compared to a specified and storedthreshold shape, e.g., a circle having a roundness tolerance of a radiusof +/−1 millimeter from a center the pixel grouping. The brightness maybe compared to one or more specified and stored brightness thresholds,e.g., brighter than 1200 lux and darker than 120 lux. The size may becompared to one or more specified and stored threshold sizes, e.g.,smaller than 12 millimeters and greater than 2 millimeters.

Thresholds may be predetermined and stored in a non-volatile memory ofthe computer 26. The brightness and/or size thresholds may bepredetermined based on empirical testing, e.g., analysis of image dataof one or more known liquid droplets on a surface. The computer 26 mayselect among one or more stored thresholds, e.g., based on a time ofday, amount of ambient light, analysis of a remainder of pixels in theimage data, etc. For example, in daylight liquid droplets 14 on thesurface 16 a, 16 b may be less bright than a remainder of an image, andduring night, i.e., in the absence of daylight, liquid droplet 14 on thesurface 16 a, 16 b may be brighter, e.g., from reflecting headlights ofanother vehicle, a streetlight, etc. The computer 26 may use othertechniques and processes to identify the liquid droplet 14.

The computer 26 may identify a position of the liquid droplet 14. Asused herein, a position of the liquid droplet 14 is a location of theliquid droplet 14 on a surface 16 a, 16 b as specified by a group ofpixels identified in image data as the liquid droplet 14. For example, aposition of a liquid droplet 14 may be specified by a vertical and ahorizontal location of the group of pixels identified in image data asthe liquid droplet 14, e.g., x,y coordinates or the like indicating acolumn and a row of the image data where the group of pixels identifiedas a liquid droplet 14 are located. The x,y coordinates determined for adroplet 14 in an image can then be mapped to x,y coordinates or the likefor a surface 16 a, 16 b using conventional techniques, e.g., an imagecould include a fiducial marker such as an edge, boundary, or othermarker of the surface 16 a, 16 b which could then be used to mapcoordinates from an image to coordinates, i.e., a location, of theliquid droplet 14 on a surface 16 a, 16 b,

The computer 26 may identify a time at which the liquid droplet 14 wasat the identified position. The computer 26 may store, in the memory,one or more positions of the liquid droplet 14 and associated times,e.g., tracking movement of the liquid droplet 14 over time. In FIG. 5 aposition P0 of the liquid droplet 14 is shown at a time T0 (i.e., FIG. 5illustrates a droplet 14 position P0 at current time T0); further storedpositions and associated times of the liquid droplet 14 are shown atP-1, T-1; P-2, T-2; etc., i.e., showing movement of the droplet 14 overtime.

The computer 26 may be programmed to predict a path 24 of a liquiddroplet 14 on the surface 16 a, 16 b. The path 24 of the liquid dropindicates one or more predicted future positions of the liquid droplet14. The path 24 may indicate one or more future times at which theliquid droplet 14 is predicted to be at the one or more predicted futurepositions. The path 24 may include one or more discrete position andtimes, e.g., relative to the identified position of the liquid droplet14, relative to the top and side edges of the image data, etc. In otherwords, the path 24 may be data indicating a series of positions andtimes. For example, the path 24 may include a formula or algorithm,e.g., indicating a predicted future position of the liquid droplet 14 asa function of time e.g., relative to the identified position of theliquid droplet 14, relative to the top and side edges of the image data,etc. The formula or algorithm may indicate a linear path, a curved path,and/or a combination thereof. The computer 26 may use various measuredvalues and predetermined constants determine the formula or algorithmused to predict the path 24, e.g., using Newtonian mechanics, fluiddynamics, etc. Such formula or algorithm may utilize known finiteelement or volume methods for liquid path prediction, e.g., using theLagrangian technique and/or Navier Stokes equation, and important modelparameters such as the Reynolds number for the liquid droplet 14, theWeber number for the liquid droplet 14, etc. As another example, a deepneural network 200 of the computer 26 may be trained to predict the path24 (further described below). In FIG. 5 the predicted future positionsand associated times are shown at P1, T1; P2, T2; etc.

The computer 26 may be programmed to predict the path 24 based on avelocity of the vehicle 12. Velocity of the vehicle 12 is a speed, e.g.,in miles per hour, of the vehicle 12 relative to a ground on which thevehicle 12 is supported. The velocity of the vehicle 12 may include adirection, e.g., a compass heading direction, a direction relative tothe vehicle 12, etc. The computer 26 may identify the velocity of thevehicle 12 based on information from the sensors 40, the navigationsystem 44, etc., e.g., received via the communication network 46 andindicating a wheel speed, a compass heading direction of the vehicle 12,a change in position of the vehicle 12 over time, etc.

The velocity of the vehicle 12 affects the path 24 at least in part byaffecting a relative velocity of ambient air to the vehicle 12. Such airstrikes and/or travels along the surface 16 a, 16 b and urges the liquiddroplet 14 to move along the surface 16 a, 16 b, e.g., parallel to adirection of movement, e.g., as specified by a velocity vector, of theambient air. For example, the liquid droplet 14 may be urged toward therear of the vehicle 12 when the velocity of the vehicle 12 in a vehicleforward direction.

The computer 26 may be programmed to predict the path 24 based on a windvelocity. Wind velocity is a speed, e.g., in miles per hour, of movementof ambient air relative to a ground on which the vehicle 12 issupported. The wind velocity may include a direction, e.g., a compassheading direction, a direction relative to the vehicle 12, etc. Thecomputer 26 may determine the direction of the wind velocity of thevehicle 12, e.g., by combining the compass heading direction of thevehicle 12 with the compass heading direction of the wind velocity. Thecomputer 26 may determine the wind velocity based on information fromthe sensors 40, information received from a remote computer indicatingthe wind velocity, etc. The wind velocity affects the path 24 at leastin part by affecting the relative velocity of ambient air to the vehicle12, e.g., as discussed above for the velocity of the vehicle 12.

The computer 26 may be programmed to predict the path 24 based on anincline angle 48 of the surface 16 a, 16 b. The incline angle 48 is anangle of the surface 16 a, 16 b relative to a horizon, i.e., relative toa level horizonal axis 50 (shown in FIG. 2). The incline angle 48controls a normal force applied to the liquid droplet 14 by the surface16 a, 16 b. For example, when the incline angle 48 is 0 degrees, thenormal force urges the liquid droplet 14 directly upward, e.g., directlyopposite the force of gravity, inhibiting downward movement of theliquid droplet 14. As another example, when the incline angle 48 is 90degrees, the normal force is perpendicular to the force of gravity anddoes not counteract any of the force of gravity. The incline angle 48,and the direction and magnitude of the normal force, may be apredetermined constant based on a design of the vehicle 12, e.g., afixed angle of the windshield 20 and/or the lens 22, that is stored inthe memory.

The computer 26 may be programmed to predict the path 24 based on anacceleration of the vehicle 12. The acceleration of the vehicle 12 is arate of change of velocity of the vehicle 12, e.g., increasing ordecreasing speed, changing a heading direction, etc. The computer 26 maydetermine the acceleration of the vehicle 12 based on information fromthe sensors 40, navigation system 44, etc., e.g., received via thecommunication network 46. The acceleration affects the path 24 of thevehicle 12 based on a difference in momentum between the liquid droplet14 and the surface 16 a, 16 b when the vehicle 12 is accelerating. Forexample, when the braking system 34 actuates to decelerate the vehicle12, the momentum of the liquid droplet 14 relative to the deceleratingvehicle 12 may urge the liquid droplet 14 toward the front of thevehicle 12.

The computer 26 may be programmed to predict the path 24 based on a sizeof the liquid droplet 14. The size of the liquid droplet 14 may be alength of the liquid droplet 14 along the surface 16 a, 16 b, a width ofthe liquid droplet 14 along the surface 16 a, 16 b, a diameter of theliquid droplet 14, etc. The computer 26 may determine the size of theliquid droplet 14 based on information from the optical sensor 18. Forexample, the computer 26 may determine the length, width, diameter,etc., of the group of pixels identified as the liquid droplet 14.

The size of the liquid droplet 14 alters the effect of other factors onthe path 24 of the liquid droplet 14. For example, a larger liquiddroplet may be more likely to travel along the surface 16 a, 16 b whenthe vehicle 12 accelerates, in response to ambient air movement relativeto the vehicle 12, etc., as compared to a smaller liquid droplet. Forexample, the larger droplet may have increased momentum that affectsmovement from acceleration of the vehicle 12, increased drag thataffects the magnitude of force on the liquid droplet 14 from ambient airmovement, etc.

The computer 26 may be programmed to predict the path 24 based on ahydrophobicity of the surface 16 a, 16 b. Hydrophobicity of the surface16 a, 16 b is an amount of absorption/repulsion of the liquid droplet 14on the surface 16 a, 16 b. Hydrophobicity of a surface may be referredto as wettability. Hydrophobicity as is known is a unitless quality andis quantified with various scales that provide relative indications ofhydrophobicity, e.g., interface scale, octanol scale, andoctanol-interface scale. The hydrophobicity may be indicated by ameasured and/or calculated contact angle of a liquid droplet 14 with thesurface 16 a, 16 b. The higher the hydrophobicity the less forcerequired to move the liquid droplet 14 along the surface 16 a, 16 b. Thehydrophobicity may depend on a texture of the surface 16 a, 16 b, acoating on the surface 16 a, 16 b, etc. The hydrophobicity may be usedto determine a constant for use when determining the path 26, e.g.,determined based on empirical testing. In other words, thehydrophobicity may be used in combination with other factors, e.g., aliquid droplet 14 may be more likely to travel along a surface 16 a, 16b with a relatively higher hydrophobicity when the vehicle 12accelerates, in response to ambient air movement relative to the vehicle12, etc., as compared to a liquid droplet 14 on a surface 16 a, 16 bwith a relatively lower hydrophobicity. The computer 26 may determinethe hydrophobicity, e.g., based on a detected and/or determined contactangle of the liquid droplet 12 on the surface 16 a, 16 b. The computer26 may use the detected and/or determined contact angle to select aconstant for determining the path 26, e.g., with a look up table or thelike.

The computer 26 may be programmed to predict the path 24 based on thecleaning system 28. For example, actuation of the cleaning system 28 maygenerate air flow, e.g., from the nozzle 42 and across the surface 16 a,16 b. For example, the computer 26 may store a force and direction ofsuch force applied to the liquid droplet 14 when the cleaning system 28is actuated. The computer 26 may store multiple forces and directions,e.g., a direction for each nozzle 42 and various forces depending onvarious air flow rates out of the nozzle 42. The computer 26 may storevarious forces based on a position of the liquid droplet 14 relative tothe nozzle 42, e.g., air from the nozzle 42 may apply greater force tothe liquid droplet 14 when the droplet is closer to the nozzle 42 ascompared to when the liquid droplet 14 is further away.

The computer 26 may be programmed to predict the path 24 based on storeddata indicating one or more previous positions of the liquid droplet 14.For example, the stored previous positions of the liquid droplet 14(P-1, T-1, P-2, T-2, etc.) may indicate that the liquid droplet 14 ismoving across the surface 16 a, 16 b in a certain direction and at acertain speed. The direction may be indicated by one or more vectors,e.g., extending from P-3, T-3 to P-2, T-2, from P-2, T-2 to P-1, T-1,etc. The speed may be indicated by a length of such vectors, e.g., whenthe positions are captured at regular time intervals. In other words,the computer 24 may determine such speed and direction based on thechange in positions and associated change in times of the stored data.The computer 26 may extrapolate additional positions at future times topredict the path 24, e.g., the computer 24 may determine a best fitcurve to the stored positions and the predicted positions may be alongsuch best fit curve.

The computer 26 may be programmed to predict a path 24 with a machinelearning programming, e.g., a neural network, such as a deep neuralnetwork 200 (shown in FIG. 4). The DNN 200 can be a software programthat can be loaded in memory and executed by a processor included in thecomputer 26, for example. The DNN 200 can include n input nodes 205,each accepting a set of inputs i (i.e., each set of inputs i can includeon or more inputs x). The DNN 200 can include m output nodes (where mand n may be, but typically are not, a same number) provide sets ofoutputs o₁ . . . o_(m). The DNN 200 includes a plurality of layers,including a number k of hidden layers, each layer including one or morenodes 205. Each layer may consist of a specific type such as fullyconnected, convolutional, dropout, pooling, softmax, etc. The nodes 205are sometimes referred to as artificial neurons 205, because they aredesigned to emulate biological, e.g., human, neurons. A neuron block 210illustrates inputs to and processing in an example artificial neuron 205i. A set of inputs x₁ . . . x_(r) to each neuron 205 are each multipliedby respective weights w_(i1) . . . w_(ir), the weighted inputs thenbeing summed in input function Σ to provide, possibly adjusted by a biasb_(i), net input a_(i), which is then provided to activation function ƒ,which in turn provides neuron 205 i output y_(i). The activationfunction ƒ can be a variety of suitable functions, typically selectedbased on empirical analysis. The respective neurons 205 may be feedforward or recurrent, e.g., long short-term memory (LSTM) units.

A set of weights w for a node 205 together are a weight vector for thenode 205. Weight vectors for respective nodes 205 in a same layer of theDNN 200 can be combined to form a weight matrix for the layer. Biasvalues b for respective nodes 205 in a same layer of the DNN 200 can becombined to form a bias vector for the layer. The weight matrix for eachlayer and bias vector for each layer can then be used in the trained DNN200. Training may be an iterative operation. In one example, thecomputer 180 may be programmed to perform an iterative training until anerror, i.e., a difference between an expected output (based on trainingdata e.g., obtained from simulation or experimentation) relative to anoutput from the trained DNN 200, is less than a specified threshold orloss, e.g., 10%.

The DNN 200 can be trained with inputs including velocity of the vehicle12, acceleration of the vehicle 12, wind velocity, an incline angle(s)48 of surface(s) 16 a, 16 b, a size of the liquid droplet 14, a positionof the liquid droplet 14 (P0, T0), stored data indicating one or moreprevious positions of the liquid droplet 14 (P-1, T-1; P-2, T-2; etc.),the hydrophobicity of the surface 16 a, 16 b, actuation of the cleaningsystem 28, etc., and to output a predicted path of the liquid droplet14, including predicted positions and associated times (P1, T1; P2, T2;etc.). The DNN 200 can be trained with ground truth data, i.e., dataabout a real-world or baseline condition or state, such as vehicle andwind velocities, surface incline angles, liquid droplet sizes, surfacehydrophobicity, air temperature, humidity, cleaning system actuation,etc. Weights w can be initialized by using a Gaussian distribution, forexample, and a bias b for each node 205 can be set to zero. Training theDNN 200 can including updating weights and biases via conventionaltechniques such as back-propagation with optimizations. Data can beassociated with paths for training the DNN 200, i.e., known paths ofliquid droplets may be associated with the input ground truth data.

Once the DNN 200 is trained, the computer 26 can input the velocity ofthe vehicle 12, the acceleration of the vehicle 12, the wind velocity,the incline angle 48 of the surface 16 a, 16 b, the size of the liquiddroplet 14, the position of the liquid droplet 14, stored dataindicating one or more previous positions of the liquid droplet 14, thehydrophobicity of the surface 16 a, 16 b, and actuation of the cleaningsystem 28 and can output a predicted path 24 of the liquid droplet 14.

The computer 26 may be programmed to identify an amount of time for theliquid droplet 14 to reach a future position. The future position is aspecific position along the path 24. For example, the future positionmay be at an edge, i.e., an outer perimeter, of the pixels in the imagedata captured by the optical sensor 18. When the liquid droplet 14 is atthe edge (i.e., substantially out of the field of view 38) the liquiddroplet is unlikely to interfere with image data collected by theoptical sensor 18. In other words, when the liquid droplet is at theedge, a portion of the liquid droplet 14 will be out of the field ofview 38, and the path 24 may indicate that the liquid droplet 14 willmove out of the field of view 38. As another example, the futureposition may be relative to other objects 36, such as cars, pedestrians,etc., identified by the computer 26 in the image data. The computer 26may compare the predicted path 24 with the future position, where thepredicted path 24 indicates how long it is predicted for the liquiddroplet 14 to reach such position.

The computer 26 may be programmed to determine the amount of time toreach the future position is greater than a threshold amount of time.The threshold amount of time may be predetermined and stored in thememory of the computer 26. The threshold amount of time may bedetermined based on data collection requirements for the optical sensor18, e.g., the threshold amount of time may be an amount of time thecomputer 26 may operate the vehicle 12 based on information from theoptical sensor 18, e.g., while an amount of data indicating anenvironment around the vehicle 12 is limited by the liquid droplet 14.For example, how long the computer 26 may operate the vehicle 12 whilethe liquid droplet 14 obscures detection of a secondary vehiclerepresented by pixels in the image data.

The computer 26 may be programmed to determine that the path 24 is froma low priority area 52 on the surface 16 a, 16 b to a high priority area54 on the surface 16 a, 16 b, and vice versa. The high priority area 54is an area within the field of view 38 that has a higher probability ofcontaining detectable objects 36 that may interfere with operation ofthe vehicle 12 as compared to the low priority area 52. For example,when the field of view 38 of the optical sensor 18 is in the vehicleforward direction, the high priority area 54 may be at a bottom half andat a center of the image data and/or field of view 38. The center of theimage data is likely where a road on which the vehicle 12 is travelingwould be located. Objects 36 in the road would be more likely tointerfere with operation of the vehicle 12 than objects 36 to the sideof the road. Objects 36 identified in the image data are likely to becloser to the vehicle 12 the lower they are in the image data, andtherefor for more likely to interfere with navigation of the vehicle 12.Objects 36 above the bottom half of the image data are likely above thehorizon, e.g., elevated above the roadway and not likely to interferewith navigation of the vehicle 12.

The high priority area 54 and low priority area 52 may be a fixed areain the field of view 38 and stored in the memory. The computer 26 maydetermine the high priority area 54 and low priority area 52, e.g.,based on stored data indicating previously detected objects, based ondetected edges of a lane of travel and/or a roadway, where the edgesenclose the high priority area 54, based on a detected horizon where thehigh priority area 54 is below the horizon, etc. For example, movingobjects may have a higher priority that non-moving objects, objects 36moving toward interference with operation of the vehicle 12 may have ahigher priority than objects 36 moving away from interference withnavigation of the vehicle 12, etc.

The computer 26 may break down the field of view 38 and image data intomore discrete levels of probability of having objects 36 that mayinterfere with operation of the vehicle 12, e.g., very low, low,medium-low, medium, medium high, high, and very high.

The computer 26 may be programmed to actuate one or more vehiclecomponents based on the path 24. The vehicle components are electrical,electromechanical, hydraulic, or other components of the vehicle 12 thatmay be actuated by the computer 26. Example vehicle components andactuations include: actuating the steering system 32 to change asteering angle of the vehicle 12, actuating the propulsion system 30 tochange the speed of the vehicle 12, actuating the braking system 34 toslow the vehicle 12, actuating the cleaning system 28 to clean thesurface 16 a, 16 b, etc. The computer 26 may actuate vehicle componentsby transmitting one or more instructions via the communication network46.

The computer 26 may be programmed to the actuate propulsion system 30,the steering system 32, and the braking system 34 based on the path 24.For example, the computer 26 may actuate the steering system 32 tooperate the vehicle 12 closer to a right side or a left side of the laneto change the location of priority areas in the field of view 38relative to the path 24, e.g., by moving the position of the detectedlane within the field of view 38. Changing the location of the priorityareas may make it such that the path 24 and/or liquid droplet 14 arechanged from being in a high priority area 54, e.g., covering a portionthe lane, to a low priority area 52, e.g., next to the lane. As anotherexample, the computer 26 may use the path 24 to predict when one or moreobjects 36, such as other vehicles, detected in the field of view 38,may be obscured by the liquid droplet 14. When such objects 36 areobscured by the path 24 they may not be detectable based on image datafrom the optical sensor 18. The computer 26 may be programmed to operatethe vehicle 12, e.g., in the autonomous mode and/or semi-autonomousmode, as if the detection of the object 36 was not lost. In other words,the computer 26 may operate the vehicle 12 based on an assumption thatthe object 36 detected in the path 24 is still in a same position,moving with at a same trajectory, etc., when the liquid droplet 14covers the object 36 as the liquid droplet 14 travels along the path 24.The computer 26 may further analyze image data along the path 24 tore-acquire detection of the object 36 after the liquid droplet 14 hasmoved further along the path 24 and no longer covers the object 36.

The computer 26 may be programmed to actuate the cleaning system 28,e.g., to the on state, based on the path 24. The computer 26 may actuatethe cleaning system 28 by transmitting an instruction via thecommunication network 46 indicating the on-state in which air isprovided to the nozzles 42 and blows across the surface 16 a, 16 b.Blowing air across the surface 16 a, 16 b may change the path 24.Additionally/alternatively, the computer 26 may transmit an instructionindicating actuation of wipers of the cleaning system 28.

The computer 26 may be programmed to predict a second path 24 of theliquid droplet 14 after actuating the cleaning system 28, e.g., to theon state. After the cleaning system 28 is actuated may be after thecleaning system 28 is actuated to the on state, e.g., while air is beingprovided to the surface 16 a, 16 b from the nozzles 42. After thecleaning system 28 is actuated may be after the cleaning system 28 isactuated to the on state, e.g., to blow or wipe the surface 16 a, 16 b,and then is actuated to an off state, e.g., to stop blowing or wipingthe surface 16 a, 16 b. The computer 26 may also evaluate the effects ofthe actuating the cleaning system 28, e.g. air pressure, on thepredicted second path.

The computer 26 may be programmed to wait an amount of time afterpredicting the path 24, and then update one or more parameters of theprediction based on a comparison of a subsequently identified positionof the liquid droplet 14 with the path 24. Updating the parameters basedon the comparison improves accuracy of subsequent predicted paths 24.The amount of time may be predetermined, e.g., 500 milliseconds, andstored in the memory of the computer 26. The parameters of theprediction are one or more values used by the computer 26 whendetermining the predicted path 24 of the liquid droplet 14. For example,the computer 26 may update the constant indicating the hydrophobicity ofthe surface 16 a, 16 b, e.g., such that the position of previouslypredicted path 24 matches the subsequently identified position of theliquid droplet 14.

Process

FIG. 6 is a process flow diagram illustrating an exemplary process 500for operating the system 10. The process 500 begins in a block 505 wherethe computer 26 receives data from the optical sensor 18, the sensors40, the navigation system 44, etc., e.g., via the communication network46. The computer 26 may receive such data substantially continuously orat time intervals, e.g., every 50 milliseconds. The computer 26 maystore the data, e.g., on the memory.

At a block 510 the computer 26 identifies a liquid droplet 14 on thesurface 16 a, 16 b. The computer 26 may identify the liquid droplet 14,and the position of such liquid droplet 14 (including multiple positionsover time), based on image data received from the optical sensor 18 viathe communication network 46, e.g., as described herein. The computer 26may store data indicating the detected position(s) of the identifiedliquid droplet 14. The computer 26 may continue to identify and storedetected position(s) of the liquid droplet 14 throughout the process500.

Next at a block 515 the computer 26 predicts a path 24 of the liquiddroplet 14 identified on the surface 16 a, 16 b at the block 510. Thecomputer 26 may predict the path 24 of the liquid droplet 14 based onone or more of the acceleration of the vehicle 12, the velocity of thevehicle 12, the wind velocity, the incline angle 48 of the surface 16 a,16 b, the size of the liquid droplet 14, the hydrophobicity of thesurface 16 a, 16 b, and/or on stored data indicating one or moreprevious detected positions of the liquid droplet 14 e.g., as describedherein.

Next at a block 520 the computer 26 determines whether the path 24indicates the liquid droplet 14 is predicted to move from a highpriority area 54 to a low priority area 52 in the field of view 38 ofthe optical sensor 18, e.g., as described herein. When the computer 26determines the liquid droplet 14 is predicted to move from the highpriority area 54 to the low priority area 52 the process 500 moves to ablock 525. When the computer 26 determines the liquid droplet 14 is notpredicted to move from the high priority area 54 to the low priorityarea 52 the process 500 moves to a block 535.

At the block 525 the computer 26 identifies an amount of time for theliquid droplet 14 to reach the low priority area 52, e.g., based on thepredicted path 24 and as described herein.

Next at a block 530 the computer 26 determines whether the amount oftime identified in the block 525 is greater than a threshold amount oftime, e.g., as described herein. When the computer 26 determines theamount of is greater than the threshold the process 500 moves to a block540. When the computer 26 determines the amount of time is not greaterthan the threshold amount of time the process moves to a block 545.

Next at a block 535 the computer 26 determines whether the path 24indicates the liquid droplet 14 is predicted to move from a low priorityarea 52 to a high priority area 54 in the field of view 38 of theoptical sensor 18, e.g., as described herein. When the computer 26determines the liquid droplet 14 is predicted to move from the lowpriority area 52 to the high priority area 54 the process 500 moves tothe block 540. When the computer 26 determines the liquid droplet 14 isnot predicted to move from the low priority area 52 to the high priorityarea 54 to the process 500 moves to the block 545.

At the block 540 the computer 26 actuates one or more vehiclecomponents, e.g., as described herein. For example, the computer 26 maytransmit an instruction to the cleaning system 28 to actuate to the onstate to blow air across the surface 16 a, 16 b. After an amount oftime, e.g., 2 seconds, the computer 26 may actuate the cleaning system28 to the off state. As another example, the computer 26 may actuate thepropulsion system 30, the steering system 32, and/or braking system 34based on the path 24, e.g., as described herein. After the block 540 theprocess may return to the block 515, e.g., to again predict the path 24of the liquid droplet 14. Alternatively, the process 500 may end.

At the block 545 the computer 26 waits an amount of time, e.g., asdescribed herein. Next at a block 550 the computer 26 identifies aposition of the liquid droplet and updates one or more parameters of theprediction, e.g., of the algorithm used to determine the path 24, basedon a comparison of the position of the liquid droplet 14 identified inthe block 550 with the path 24 predicted in the block 515. After theblock 550 the process 500 may end. Alternatively, the process 500 mayreturn to the block 510.

Conclusion

With regard to the process 500 described herein, it should be understoodthat, although the steps of such process 500 have been described asoccurring according to a certain ordered sequence, such process 500could be practiced with the described steps performed in an order otherthan the order described herein. It further should be understood thatcertain steps could be performed simultaneously, that other steps couldbe added, or that certain steps described herein could be omitted. Inother words, the description of the process 500 herein is provided forthe purpose of illustrating certain embodiments and should in no way beconstrued so as to limit the disclosed subject matter.

Computing devices, such as the computer 26, generally includecomputer-executable instructions, where the instructions may beexecutable by one or more computing devices such as those listed above.Computer-executable instructions may be compiled or interpreted fromcomputer programs created using a variety of programming languagesand/or technologies, including, without limitation, and either alone orin combination, Java™, C, C++, Visual Basic, Java Script, Python, Perl,etc. Some of these applications may be compiled and executed on avirtual machine, such as the Java Virtual Machine, the Dalvik virtualmachine, or the like. In general, a processor (e.g., a microprocessor)receives instructions, e.g., from a memory, a computer-readable medium,etc., and executes these instructions, thereby performing one or moreprocesses, including one or more of the processes described herein. Suchinstructions and other data may be stored and transmitted using avariety of computer-readable media.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory (e.g., tangible) medium thatparticipates in providing data (e.g., instructions) that may be read bya computer (e.g., by a processor of a computer). Such a medium may takemany forms, including, but not limited to, non-volatile media andvolatile media. Non-volatile media may include, for example, optical ormagnetic disks and other persistent memory. Volatile media may include,for example, dynamic random access memory (DRAM), which typicallyconstitutes a main memory. Such instructions may be transmitted by oneor more transmission media, including coaxial cables, copper wire andfiber optics, including the wires that comprise a system bus coupled toa processor of a computer. Common forms of computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, any other magnetic medium, a CD-ROM, DVD, any otheroptical medium, punch cards, paper tape, any other physical medium withpatterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any othermemory chip or cartridge, or any other medium from which a computer canread.

In some examples, system elements may be implemented ascomputer-readable instructions (e.g., software) on one or more computingdevices (e.g., servers, personal computers, computing modules, etc.),stored on computer readable media associated therewith (e.g., disks,memories, etc.). A computer program product may comprise suchinstructions stored on computer readable media for carrying out thefunctions described herein.

The disclosure has been described in an illustrative manner, and it isto be understood that the terminology which has been used is intended tobe in the nature of words of description rather than of limitation. Manymodifications and variations of the present disclosure are possible inlight of the above teachings, and the disclosure may be practicedotherwise than as specifically described.

What is claimed is:
 1. A system, comprising: a processor; and a memorystoring instructions executable by the processor to: predict a path of aliquid droplet on a surface; and then, actuate one or more vehiclecomponents based on the path.
 2. The system of claim 1, the instructionsfurther including instructions to determine that the path is from a lowpriority area on the surface to a high priority area on the surface,wherein the high priority area has a higher probability of containingdetectable objects that may interfere with operation of a vehicle ascompared to the low priority area.
 3. The system of claim 1, theinstructions further including instructions to predict the path based onat least one of a vehicle velocity, a wind velocity, an incline angle ofthe surface, a vehicle acceleration, a size of the liquid droplet, and ahydrophobicity of the surface.
 4. The system of claim 1, theinstructions further including instructions to predict the path based onstored data indicating one or more previous positions of the liquiddroplet.
 5. The system of claim 1, the instructions further includinginstructions to actuate the one or more vehicle components by actuatinga cleaning system.
 6. The system of claim 5, the instructions furtherincluding instructions to predict a second path of the liquid dropletafter actuating the cleaning system.
 7. The system of claim 1, theinstructions further including instructions to actuate the one or morevehicle components by actuating at least one of a steering system, abraking system, and a propulsion system.
 8. The system of claim 1, theinstructions further including instructions to wait an amount of timeafter predicting the path, and then update one or more parameters of theprediction based on a comparison of a position of the liquid dropletwith the path.
 9. The system of claim 1, further comprising an opticalsensor defining a field of view and in communication with the processor,wherein the surface is in the field of view.
 10. The system of claim 1,the instructions further including instructions to identify an amount oftime for the liquid droplet to reach a future position, and to determinethe amount of time to reach the future position is greater than athreshold amount of time.
 11. The system of claim 1, wherein the surfaceis on one of a lens and a windshield.
 12. A method, comprising:predicting a path of a liquid droplet on a surface; and then, actuatingone or more vehicle components based on the path.
 13. The method ofclaim 12, wherein the path is predicted based on at least one of avehicle velocity, a wind velocity, an incline angle of the surface, avehicle acceleration, a size of the liquid droplet, and a hydrophobicityof the surface.
 14. The method of claim 12, further comprising waitingan amount of time after predicting the path, and then updating one ormore parameters of the prediction based on a comparison of a position ofthe liquid droplet with the path.
 15. The method of claim 12, furthercomprising identifying an amount of time for the liquid droplet to reacha future position, and to determine the amount of time to reach thefuture position is greater than a threshold amount of time.
 16. Themethod of claim 12, wherein actuating one or more vehicle componentsincludes actuating a cleaning system, and further comprising predictinga second path of the liquid droplet after actuating the cleaning system.17. The method of claim 12, wherein the path is predicted based onstored data indicating one or more previous positions of the liquiddroplet.
 18. A system, comprising: means for detecting a liquid dropleton a surface; means for predicting a path of the liquid droplet on thesurface; and means for moving the liquid droplet relative to the surfacebased on the path.
 19. The system of claim 18, further comprising meansfor navigating a vehicle based on the path.
 20. The system of claim 18,further comprising means for determining that the path is from a lowpriority area on the surface to a high priority area on the surface,wherein the high priority area has a higher probability of containingdetectable objects that may interfere with operation of a vehicle ascompared to the low priority area.